While there are many advantages to connecting internal computer networks to external networks, there is also the inherent danger of attacks. In order to minimize the danger of external attacks it is important to be able to detect threats as quickly as possible, preferably in real time, such that they can be blocked or otherwise disrupted before any harm is done.
While real time threat analysis has been a goal in network security, it has been an unattainable goal for several reasons. Network security alerting has been achieved by deploying a variety of security detection devices, also know as point devices, which analyze data flow in a network to identify threats and to generate alerts when threats are found. These devices are limited because they have a narrow view of the world in that they only see data and traffic which is either in-band (i.e. data actually flowing through the device) or that can be promiscuously sniffed from the wire (i.e. data that passes near the device when it is switched into promiscuous mode). In either case subnet topologies limit the view such devices have.
The known devices are further limited in that they attempt to identify threats based on the signatures of known and established attack vectors. In other words, these devices are only capable of identifying attacks from known attackers or attacks that are copies of attacks that have occurred before; they are unable to identify threats based on non-signature based traffic. The current devices can only flag well known and understood threats using very deterministic procedures, and while this may be effective in countering old attacks, it is of no use in discovering novel or non-signature based threats. Accordingly, network systems are left wide open to such new attacks which are capable of causing enormous amounts of damage.
As an alternative to the above discussed threat detection devices, network security systems also employ protection devices such as firewall and proxies. These protection devices log data streams that also contain threat information. However, the data logs that are generated are of little or no practical value because of their size; the rapid data flow through these devices results in excessively large data logs that were heretofore unwieldy for use in any sort of real time data analysis. Due to the size of the data logs and the volumes of seemingly irrelevant data it has been impractical to search through these logs for meaningful data to detect threat conditions.